AETG vs. Man: an Assessment of the Effectiveness of Combinatorial Test Data Generation

This paper reports on an industrial study of the effectiveness of test data generation. In the literature on the automatic generation of test data a number of techniques stand out as having received a significant amount of interest. One area that has achieved considerable attention is the use of combinatorial techniques to construct data adequate test sets that ensure all pairs, triples etc. of input variables are included in at least one test vector. There has been some systematic evaluation of the technique as applied to unit testing and, while there are indications that the technique can be effective, very little work has been performed using industrial code. Moreover, there has been no comparison of effectiveness of the technique for unit testing compared with tests that are generated by hand. In this paper we apply random and combinatorial (AETG) techniques to a number of functions drawn from industrial code with known faults and existing unit test suites. Results indicate that for simple cases combinatorial techniques can be as effective as the human-generated test, but there are instances associated with complex code where the technique performs poorly—but no worse than randomly generated data.